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Abstract 

This  paper  describes  the  collaboration  of  Brigham  and  Women’s  Hospital  and 
Regenstrief  Institute  to  develop  a  computerized  adverse  drug  event  (ADE) 
monitor  using  electronic  medical  records  from  outpatient  practices.  We  describe 
the  steps  involved  in  ADE  monitor  development  and  rule  validation  at  large 
outpatient  practices  at  Boston  and  Indianapolis.  The  final  standard  rule  set 
adopted  by  both  practice  sites  are  currently  being  used  to  test  the  impact  of  basic 
and  advanced  decision  support  on  ADE  rates. 

The  rules  used  by  the  ADE  monitor  derive  from  coded  medication  names  and 
laboratory  results,  as  well  as  text  from  clinician  notes  contained  within  the 
electronic  medical  record  systems.  The  nontext  rules  are  subdivided  into  five 
categories:  medication,  laboratory,  medication-laboratory,  ICD-9  codes,  or 
miscellaneous.  Rules  target  various  diagnostic  and  laboratory  abnormalities 
caused  by  a  broad  range  of  outpatient  medications  commonly  used  in  primary 
care.  Text-based  rules  were  developed  for  certain  medications  by  linking  the 
medications  with  symptoms  (ADE)  that  are  often  associated  with  short-  or  long¬ 
term.  The  rules  were  run  on  4  months  of  data  at  both  sites,  possible  ADEs  were 
identified  and  validated  by  chart  review,  and  the  positive  predictive  values  of  each 
rule  were  calculated.  We  found  that  clinically  based  rule  sets  can  be  developed 
and  implemented  at  different  outpatient  settings  using  distinct  information 
systems  to  identify  ADEs  related  to  commonly  prescribed  outpatient  medications. 


Introduction 

Adverse  drug  events  (ADEs)  contribute  to  overall  morbidity  and  mortality. 
Phillips  et  al. 1  have  estimated  that  as  many  as  7,000  people  die  each  year  due  to 
medication  errors  both  outside  and  inside  the  hospital.  It  has  also  been  reported  in 
an  academic  teaching  hospital  that  the  rates  are  6.5  ADEs  per  100  admissions.2  In 
another  study,  a  total  ADE  event  rate  was  found  to  be  27.4  per  100  patients  in  the 
outpatient  setting.3  These  studies  show  that  ADEs  are  common  and  many  are 
preventable  in  inpatient  and  outpatient  settings. 

Previous  studies  have  demonstrated  the  use  of  ADE  detection  monitors  to 
identify  ADEs2’ 4’ 5  and  possibly  prevent  further  hann  to  the  patient.  Jha  et  al. 
demonstrated  that  the  ADE  monitor  that  they  used  required  substantially  fewer 
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person-hours  than  did  chart  review  and  thus  would  require  fewer  resources  to 
monitor  ADEs.  They  also  felt  that  it  was  the  most  practical  method  for  ongoing 
quality  assessment.4  Another  study  has  shown  the  ability  to  use  an  ADE  monitor 
in  inpatient  pharmacy  practice  to  provide  recommendations  to  prescribing 
clinicians  on  a  daily  basis.6  In  a  study  by  Honigman  et  al.  an  outpatient  ADE 
monitor  was  developed.  The  utility  of  multiple  search  methods  was  demonstrated 
within  their  ADE  monitor  and  the  use  of  an  electronic  medical  record  led  to  the 
ability  to  include  symptom  tenns  as  part  of  their  methodology.7  Yet  another  study 
using  an  ADE  monitor  was  undertaken  in  the  ambulatory  setting  using  a  similar 
set  of  rules.  These  studies  all  demonstrate  the  potential  feasibility  of 
implementing  ADE  monitors  in  inpatient  and  outpatient  settings. 

While  computerized  monitors  are  not  yet  widely  used,  they  offer  an  efficient 
approach  for  monitoring  the  frequency  of  ADEs  on  an  ongoing  basis  and  the 
Health  Care  Financing  Administration  is  considering  mandating  them.9  These 
monitors  need  to  be  evaluated  and  refined  further;  in  particular,  it  is  important  to 
quantify  the  yield  of  collecting  these  data  on  an  ongoing  basis.  In  addition, 
refinement  of  the  rules  used  in  these  monitors  is  necessary  in  the  outpatient 
setting  because,  unlike  hospitalized  patients,  outpatient  clinical  infonnation  is 
often  not  computerized.  If  computerized  data  are  available,  key  pieces  of 
information  may  reside  in  disparate  systems.  Therefore,  development  of 
automated  systems  to  identify  ADEs  in  these  settings  is  more  difficult.  As 
computerized  outpatient  records  become  more  common,  further  work  needs  to  be 
done  to  adequately  obtain  infonnation  from  nontraditional  sources,  such  as  clinic 
notes. 

Our  goal  was  to  update  outpatient  rule  sets  previously  described  in  Honigman 
et  al.7  and  demonstrate  use  in  two  different  practice  settings  with  two  different 
electronic  medical  records.  Regenstrief  Institute  and  Brigham  and  Women’s 
Hospital  collaborated  to  create  and  test  an  outpatient  ADE  monitor,  using  data 
from  electronic  sources.  The  ultimate  goal  was  to  have  a  well-tested  monitor  that 
can  eventually  be  used  for  measurement  purposes  such  as  medication-safety 
related  interventions. 


Settings 

Brigham  and  Women’s  Hospital,  Boston 

Brigham  and  Women’s  Hospital  is  a  700-bed  institution  that  provides 
primary,  secondary,  and  tertiary  care  in  Boston,  and  is  affiliated  with  Harvard 
Medical  School  and  Partners  Healthcare  Systems.  The  hospital  has  approximately 
170  clinicians  who  serve  as  primary  care  physicians  in  various  practice  sites, 
including  hospital-based  clinics,  community-based  clinics,  and  neighborhood 
health  centers.  Since  2000,  all  primary  care  physicians  (PCPs)  have  used  an 
electronic  ambulatory  record  called  the  Longitudinal  Medical  Record  (LMR)  to 
store  information  on  the  outpatients  such  as  demographics,  problem  lists, 
medication  lists,  allergy  lists,  test  results,  and  visit  notes.  LMR  currently  is  online 
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in  all  primary  care  sites,  giving  clinicians  access  to  a  patients’  electronic  medical 
record  with  Web-based  browser  functionality  and  user-friendly  design.  Visit  notes 
can  be  entered  into  a  patient’s  medical  record  by  the  clinician,  although  the 
majority  of  providers  still  dictate  encounter  notes  to  be  transcribed.  In  addition, 
prescriptions  can  be  printed  electronically. 

Regenstrief  Institute,  Indianapolis 

Indiana  University  Medical  Group  Primary  Care  (IUMG-PC)  is  a  physician- 
led,  primary  care  practice  that  includes  internists,  pediatricians,  family 
practitioners  and  obstetrician/gynecologists  who  deliver  care  to  patients  in  12 
office  locations  located  in  central  Indiana.  The  practice  is  affiliated  with  the 
Indiana  University  School  of  Medicine  and  Wishard  Memorial  Hospital. 
Physicians  in  6  of  the  12  IUMG-PC  primary  care  practices  use  the  Medical 
Gopher  clinical  workstation  software.  A  version  of  the  Gopher  system  has  been  in 
use  for  outpatient  order  entry  since  1984. 10  The  Gopher  provides  order  entry, 
workflow,  clinical  documentation,  results  review,  access  to  clinical  knowledge 
bases  and  linkages  to  billing  functions.  The  provider  records  the  patient’s  identity, 
allergies,  billing  diagnoses,  medications  prescribed,  tests  ordered,  referrals, 
nursing  interventions  ordered,  and  notes. 


Methods 

Development  of  ADE  monitors  have  been  described  in  various  studies,4' 7' 1 1 
and  rules  from  these  prior  studies  were  used  to  develop  our  outpatient  monitor. 
Our  goal  was  to  enhance  the  effectiveness  of  the  ADE  detection  monitor  and 
implement  it  in  two  different  electronic  medication  record  (EMR)  systems,  using 
two  major  strategies  to  detect  ADEs  in  the  outpatient  arena. 

One  detection  method  used  non-text  triggers  (or  rules)  that  search  a  patient’s 
recorded  laboratory  values  and  medication  list,  and  applies  a  set  of  logical  rules  to 
determine  a  possible  ADE  or  medication  error.  In  addition,  other  data  sources  are 
used  in  some  triggers  such  as  ICD-9  codes  for  toxicity  due  to  lithium;  CPT  codes 
for  an  esophagogastroduodenoscopy;  or  demographic  information  such  as  age. 
The  second  detection  method  also  uses  the  patient’s  medication  list,  but  uses  an 
algorithm  to  identify  the  presence  of  symptoms  by  using  symptom  concepts  in  the 
text  of  electronic  notes  and  to  determine  when  they  were  negated.  Identifying 
negations  is  important  to  distinguish  “no  nausea”  from  “nausea”  and  minimize  the 
number  of  false  positive  “hits.” 

Examples  of  symptom  terms  that  might  be  used  are  “sexual  dysfunction” 
associated  with  clonidine  or  “dizziness”  associated  with  gabapentin.  Running 
these  two  detection  programs  on  the  EMR  produces  a  number  of  “hits”  (i.e., 
possible  events).  Trained  reviewers,  using  a  manual  chart,  examined  each  hit  to 
determine  whether  (1)  it  was  an  actual  or  potential  event;  and  (2)  whether  these 
events  can  be  classified  as  an  adverse  drug  event  (ADE),  medication  error  (ME), 
medication  error  that  leads  to  an  adverse  drug  event  (ADE/ME),  or  no  event. 
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In  this  study,  ADEs  were  defined  as  events  related  to  medication  use  that 
resulted  in  patient  harm.  A  medication  error  was  defined  as  any  error  that 
occurred  in  the  medication  use  process  (including  ordering/prescribing, 
dispensing,  adherence,  and  monitoring).  Medication  errors  that  caused  an  injury 
were  called  preventable  ADEs.  Medication  errors  that  had  potential  to  cause 
injury  were  called  potential  ADEs.  “  We  also  noted  some  medication  errors  that 
we  felt  had  little  chance  of  causing  harm,  but  still  represented  an  error  in  the 
medication  use  process.  An  example  of  this  would  be  a  patient  who  has  two  active 
orders  for  two  different  proton  pump  inhibitors.  The  medication  has  a  low  adverse 
reaction  profile,  but  the  therapy  is  redundant  and  thus  an  error.  Those  MEs  that 
resulted  in  harm  to  the  patient  were  classified  as  an  AD  E/ME  (also  known  as 
preventable  ADEs). 

At  Regenstrief,  data  mangers  were  initially  used  to  screen  the  results,  which 
were  then  passed  to  a  clinical  pharmacist  or  nurse  for  full  review.  The  data 
managers  received  specialized  training  using  a  manual  developed  at  Brigham  and 
Women’s  Hospital.  The  results  were  reviewed  directly  by  a  clinical  pharmacist; 
by  clinical  year  pharmacy  students  under  the  direction  of  the  clinical  pharmacist; 
or  research  assistants  who  also  received  specialized  training.  The  clinical 
pharmacist  further  reviewed  any  ambiguous  cases  that  were  discovered  by  the 
research  assistants  or  pharmacy  students. 

Nontext  rules  logic 

To  program  the  nontext  rules  we  used  an  initial  list  of  rules  known  to  have 
been  implemented  in  hospital  based  systems.4' 1 1  Both  sites  contributed  to  the 
development  of  additional  rules,  and  they  were  added  to  the  list  to  be 
programmed.  The  new  rules  were  created,  based  on  the  experience  of  the 
clinicians  working  on  the  project,  as  well  as  evidence-based  literature  reviews.  An 
example  of  this  is  adaptation  of  Beer’s  criteria  for  determining  potentially 
inappropriate  medication  use  in  the  elderly  population:  while  some  agents  listed 
in  this  guideline  were  included,  some  were  excluded  because  the  use  of  the  agent 
was  related  to  a  specific  indication  (digoxin)  or  was  no  longer  available 
(phenylbutazone).  We  also  drew  from  Micromedex®  to  develop  a  focused  list  of 
drug-drug  interactions  to  be  incorporated  in  a  single  rule. 

The  nontext  rules  were  divided  into  five  groups,  based  upon  the  type  of  data 
used  in  the  trigger:  (1)  drug;  (2)  drug-laboratory;  (3)  laboratory;  (4)  ICD-9;  and 
(5)  miscellaneous. 

The  first  category  (drug)  involved  the  use  of  medication  data.  A  new  order  of 
an  angiotensin  receptor  blocker,  for  example,  may  signify  that  the  patient  had  an 
adverse  reaction  to  an  angiotensin  converting  enzyme  inhibitor.  Another  example 
is  the  use  of  two  medications  together,  such  as  two  currently  active  orders  for 
agents  that  contain  acetaminophen,  that  may  result  in  the  patient  receiving  an 
excessive  dose  of  acetaminophen.  The  second  category  (drug-laboratory) 
involved  linking  a  medication  with  laboratory  results,  such  as  the  use  of  a 
potassium  sparing  diuretic  and  the  presence  an  elevated  potassium  level.  The  third 
category  (laboratory)  was  the  use  of  laboratory  data  only;  for  example,  a  positive 
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Clostridium  difficile  toxin  result,  an  ALT  result  greater  than  150U/L,  and  a  serum 
phenytoin  greater  than  20  mcg/ml.  The  fourth  category  (ICD-9)  was  the  use  of 
ICD-9  codes,  such  as  “poisoning  by  agents  that  affect  the  CV  system”  or 
“poisoning  by  analgesics,  antipyretics  and  anti-rheumatics.”  (Brigham  and 
Women’s  hospital  did  not  return  any  results  when  using  the  ICD-9  codes  in  the 
outpatient  setting,  due  to  difficulties  obtaining  this  data.)  The  fifth  category 
(miscellaneous)  was  all  the  rules  that  did  not  meet  the  criteria  for  the  other 
categories;  examples  inclue  the  use  of  NSAIDs  and  a  CPT  code  for  an  EGD,  any 
female  receiving  finesteride,  any  patient  receiving  ketorolac  for  greater  than  5 
consecutive  days  or  “allergic  contact  dennatitis  from  super  bill.” 

The  rules  were  programmed  and  run  on  4  months  of  EMR  data  (January  15- 
May  15,  2001).  An  initial  analysis  was  perfonned  at  both  sites  to  determine  if 
some  of  the  rules,  as  programmed,  worked  as  well  in  the  ambulatory  setting  as 
they  did  in  the  acute  care  setting.  A  positive  predictive  value  (PPV)  was 
calculated  by  using  the  total  number  of  identified  events  divided  by  the  total 
reviewed  results  for  each  rule  (with  the  duplicate  results  produced  within  each 
trigger  subtracted  out).  A  PPV  was  calculated  for  each  rule  and  for  the  entire  set 
of  trigger  within  the  non-text  set.  A  partial  list  of  the  rules  can  be  seen  in  Table  1. 


Table  1.  Examples  of  nontext  based  rules 


Rule 

Type 

Receiving  sodium  polystyrene  sulfonate 

Drug 

Combination  of  2  drugs  together  (DDIs) 

Drug 

Duplicate  therapy  (2  active  ACE  inhibitors) 

Drug 

Duplicate  therapy  (2  active  Angiotensin  receptor  blockers) 

Drug 

Serum  theophylline  >  20  mcg/ml 

Lab 

Any  positive  "C-Diff"  toxin 

Lab 

Any  fasting  blood  glucose  or  random  blood  sugar  <  54  mg/dL 

Lab 

Any  QTc  interval  >  450  ms  and  medications  that  prolong  QTc 

Drug-lab 

Receiving  digoxin  AND  serum  potassium  <  3.5  mmol/L 

Drug-lab 

Receiving  a  HMG  CoA  reductase  inhibitor  and  a  CK  >  500 

Drug-lab 

Receiving  phenytoin  and  no  phenytoin  test  within  the  past  12 
months 

Drug-lab 

ICD-9  Delerium,  Drug  induced  ICD-9  =  292.81 

ICD-9 

ICD-9  codes  “Poisoning  by  agents  that  affect  the  CV  system” 

ICD-9 

NSAID  with  EGD,  CPT  code  =  43235,  43255,  43239 

Misc.  CPT 

Age  greater  than  65  and  receiving  a  drug  in  a  modified  Beers  list 

Misc.  age 

Any  female  receiving  finesteride 

Misc.  gender 

Any  patient  receiving  >  4100  mg  of  acetaminophen  daily  (OTC) 

Misc.  dose 

Any  patient  receiving  greater  than  5  consecutive  days  of 
ketorolac 

Misc.  duration 
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Text  rule  ADE  monitor  logic 

We  created  a  set  of  rules  based  upon  a  combination  of  medication  use  and 
symptoms  that  may  indicate  that  an  ADE  had  occurred.  These  rules  were  taken 
from  monitors  used  in  other  studies4, 7  and  modified  by  review  of  evidence-based 
literature  related  to  adverse  reactions.  An  example  of  this  would  be  the 
association  of  myopathy  with  an  HMG-CoA  reductase  inhibitor.  Although  the 
rate  of  occurrence  is  low,14  this  reaction  carries  a  high  mortality,  and  cervistatin 
associated  myopathy  was  quite  well  publicized.15  A  computer  program  was 
designed  to  identify  these  possible  ADEs  and  the  program  used  natural  language 
processing  to  identify  possible  events.  The  program  used  two  aspects  of 
Regenstrief  and  Brigham  and  Women’s  Hospital  systems:  online  physician  notes 
for  patient  encounters  and  the  patients’  medication  list.  The  program  first 
identified  visit  notes  in  the  EMR,  then  examined  the  patients’  medication  list, 
looking  back  60  days  from  the  “note  date.”  If  the  patient’s  medication  list 
contained  a  medication  that  was  involved  in  the  rule-set  within  this  60-day  time 
frame,  the  program  would  then  see  if  any  medication-symptom  matches  could  be 
made  for  that  note-date.  The  Regenstrief  Institute  had  a  slightly  different  drug 
match  algorithm  than  Brigham  and  Women’s  Hospital  systems.  This  was  because 
Regenstrief  has  more  comprehensive  medication  utilization  data  from  the 
phannacy,  and  can  tell  whether  patients  have  refilled  medications  recently. 
Therefore,  Regenstrief  used  the  actually  fill  data  at  the  pharmacy  level  in  the  prior 
60  days,  while  Brigham  and  Women’s  Hospital  systems  the  presence  of  the 
targeted  agent  in  the  active  medication  list  was  used. 

The  note  was  analyzed  using  a  text  parser  for  symptom  terms  previously 
defined  that  are  linked  to  certain  medications.  Examples  of  the  text  rules  are 
angioedema  associated  with  angiotensin  converting  enzyme  inhibitors  and 
synonyms  of  bronchospasm  (such  as  wheezing,  asthma  or  asthma  attack)  related 
to  beta-blockers.  The  monitor  was  programmed  to  ignore  symptom  terms 
appearing  within  six  words  after  a  negation  term  (“not,”  “no,”  “denies,”  “lack  of,” 
etc.).  If  both  a  negated  symptom  term  and  non-negated  symptom  tenn  (a  symptom 
term  that  did  not  have  a  negation  located  within  six  words  of  it)  were  present  in 
the  note,  the  note  was  included  to  be  reviewed.  The  medications  that  were  to  be 
studied  were  divided  into  classes,  such  as  angiotensin  converting  enzyme 
inhibitors,  beta-blockers,  and  hypoglycemic  medications  (both  oral  and  injected 
insulin).  It  was  detennined  that  the  same  symptoms  terms  might  relate  to  multiple 
classes,  and  it  was  decided  that  symptoms  terms  should  be  linked  to  clinical 
concepts.  The  symptom  term  “contusion,”  for  example,  would  be  linked  with  both 
of  the  clinical  concepts  “agranulocytosis”  and  “bleeding.”  In  long-term  use  of  an 
ADE  monitor,  this  process  would  be  easier  to  maintain  links  within  the  program 
and  decrease  the  time  needed  to  maintain  the  program.  Other  examples  of  some  of 
the  rules  in  the  text  portion  of  the  ADE  monitor  are  shown  in  Table  2. 
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Table  2.  Examples  of  text-based  rules 


Rule 

ACE  Inhibitors  AND  hyperkalemia 

Anti-seizure  meds  AND  dizziness 

Anti-depressants  group  1  AND  constipation 

Anti-depressants  group  2  (SSRI,  SNRI,  and  bupropion)  AND  anorexia 

Benzodiazepines  AND  confusion 

Beta  Blockers  AND  bradycardia 

Calcium  channel  blockers  AND  peripheral  edema 

Diuretics  AND  hypokalemia 

Hypoglycemics  AND  dizziness 

Narcotics  AND  constipation 

NSAIDs  AND  hyperkalemia 

Potassium  AND  hyperkalemia 

Central  acting  anti-adrenergic  agents  AND  sexual  dysfunction 

Nitrates  AND  hypotension 

HMG-CoA  reductase  inhibitors  AND  myopathy 

Amiodarone  AND  thyroid  abnormality 

Antipsychotic  agents  AND  sedation 

Lithium  AND  polyuria 

Steroids,  Oral  AND  ulceration/perforation 

K  Sparing  medications  AND  hyperkalemia 

A  PPV  was  calculated  for  each  rule  by  adding  the  number  of  ADEs, 
medication  errors  and  ADE/MEs  discovered  by  the  monitor,  then  dividing  that  by 
the  total  reviewed  results  for  each  rule  (with  the  duplicate  results  produced  within 
each  trigger  subtracted  out).  A  PPV  was  calculated  for  each  individual  rule  and 
for  the  entire  set  of  rules  within  the  text  set.  If  an  event  was  captured  through 
multiple  triggers,  the  event  was  counted  only  once  in  the  overall  PPV  calculation. 
In  addition,  a  calculation  was  done  to  remove  the  effect  of  the  duplicate  events 
from  the  denominator  as  well  as  the  numerator  to  more  accurately  calculate  the 
overall  PPV.  Ultimately  some  of  the  rules  were  evaluated  as  not  providing  enough 
clinical  specificity  and  were  removed  from  the  monitor.  In  addition,  if  a  trigger 
had  greater  than  20  results  and  after  review,  no  events  were  identified,  the  trigger 
was  designated  as  inactive.  Based  upon  the  PPVs,  some  rules  were  eliminated 
from  the  rules  set. 
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Results 

On  some  of  the  initial  data  runs  there  was  some  divergence  in  the  predictive 
value  for  some  rules,  such  as  the  rules  that  look  for  inadequate  monitoring 
(receiving  phenytoin  and  no  phenytoin  test  in  12  months).  To  resolve  these  issues, 
we  had  programmers  and  a  clinician  from  both  sites  meet  and  reconcile  the 
programming  methods.  We  also  created  a  process  for  comparing  the  PPVs  of 
individual  triggers  between  the  two  sites.  If  there  were  significant  differences 
between  them,  we  created  a  targeted  way  of  resolving  them.  We  would  review  the 
programming  process  at  both  sites  to  ensure  we  were  obtaining  similar  output 
from  the  rules.  We  would  also  look  at  the  quality  of  the  data;  an  example  was  our 
data  related  to  some  psychiatric  medications  indicated  that  although  the 
medication  was  on  the  active  medication  list,  the  medication  may  not  have  been 
renewed  for  3-4  years  since  psychiatry  did  not  use  LMR. 

Rules  with  low  predictive  value  were  evaluated  to  see  if  improvements  could 
be  made.  An  example  of  this  is  the  rule  related  to  the  use  of  topical  steroids, 
which  may  indicate  that  a  patient  has  had  a  topical  allergic  reaction  to  a 
medication.  Each  site  selected  50  random  rule  hits  from  the  total  results  returned 
for  chart  review  and  at  both  sites,  no  events  were  found  to  occur  within  the 
sample.  It  was  then  decided  that  we  should  create  parameters  to  perhaps  limit  the 
hits  returned,  excluding  patients  who  had  either  eczema  or  psoriasis  on  their 
electronic  problem  list.  When  these  hits  were  again  reviewed,  the  outcome  was 
the  same  at  both  sites,  that  no  events  were  identified  in  the  50  randomly  selected 
results.  Since  our  attempt  to  refine  the  rules  in  order  to  identify  events  using  this 
particular  rule  resulted  in  no  improvement,  we  felt  that  this  rule  should  be 
excluded  from  the  ADE  monitor. 

Similar  processes  were  used  for  other  rule  sets  that  had  low  predictive  value. 
For  example  the  use  of  diphenhydramine  and  naloxone,  prednisone  and 
diphenhydramine  at  the  same  visit,  and  hydroxyzine  and  prednisone;  AST  level 
greater  than  150  U/L;  platelet  level  less  than  50,000;  and  receiving  inhaled 
triamcinolone  and  a  beta-blocker  were  all  eliminated,  because  of  low  predictive 
value.  There  were  a  total  of  150  triggers  that  were  initially  developed  and  128  that 
were  used  after  excluding  triggers  with  a  low  yield.  There  were  a  total  of  213 
rules  initially  developed,  137  that  were  programmed  and  106  that  ultimately  were 
included  in  the  final  text-rule  monitor. 

While  we  attempted  to  work  toward  getting  our  ADE  rates  as  similar  as 
possible,  there  were  some  rules  that  stayed  divergent.  It  is  possible  that  this  effect 
was  due  to  the  difference  in  patient  populations,  different  systems  of  care  (e.g., 
the  presence  of  an  anticoagulation  service),  different  local  treatment  guidelines 
required  by  insurers,  or  how  data  is  captured  within  the  two  different  EMRs. 


Discussion 

The  use  of  a  computer-based  monitoring  system  to  measure  the  frequency  of 
ADEs  in  the  hospital  setting  has  been  documented  in  the  literature.4' 5' 6' 1 1  In  this 
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report,  we  describe  our  methodology  in  developing  an  ADE  monitor  in  the 
outpatient  setting  at  two  different  sites  with  two  different  electronic  medical 
records.  The  ADE  monitor  that  we  developed  at  both  Regenstrief  and  Brigham 
and  Women’s  Hospital  was  based  on  triggers  that  have  been  validated  by  other 
studies  and  enhancements  that  we  added  to  improve  predictive  value.  We 
implemented  this  monitor  across  a  broad  spectrum  of  patients  in  primary  care 
practices  to  better  understand  the  rates  of  ADEs  in  the  primary  care  setting.  The 
two  sites  worked  collaboratively  to  ensure  that  consistent  programming  of  the 
rules  was  undertaken  so  that  the  output  of  the  program  would  be  as  close  as 
possible. 

There  are  differences  in  the  monitors  that  are  used  in  different  care  settings.  In 
the  monitor  that  we  developed  in  the  ambulatory  setting  there  were  some  triggers 
that  did  not  produce  results,  although  they  had  been  useful  in  the  inpatient  setting. 
The  trigger  that  looked  at  the  use  of  topical  steroids  is  one  example.  This  trigger  is 
still  currently  being  used  in  the  inpatient  ADE  monitor  at  the  Brigham  and 
Women’s  Hospital.  Conversely  we  developed  triggers  that  did  not  have  analogous 
counterparts  in  the  inpatient  setting,  such  as  a  trigger  that  looks  at  the  use  of 
phenytoin  and  the  patient  having  no  phenytoin  level  during  the  past  12  months. 
Since  most  hospitalizations  are  of  a  shorter  duration,  this  trigger  would  not  be 
effective  in  discovering  events  in  the  inpatient  setting,  but  is  important  in 
outpatient  settings. 

In  the  future  it  might  be  possible  to  extend  this  type  of  ADE  monitor  to  the 
sub-acute  setting  (rehabilitation  and  nursing  home  populations);  however, 
validation  and  expansion  of  the  triggers  will  be  necessary  in  order  to  optimize  the 
monitor  in  that  setting.  While  this  goal  is  important,  the  collection  of  the  data  in 
this  setting  may  be  problematic  as  most  of  these  institutions  would  be  unlikely  to 
have  the  information  technology  infrastructure  available  to  create  an  ADE 
monitor  similar  to  the  one  described  here. 

With  the  total  number  of  ambulatory  prescriptions  being  filled  in  the  United 
States  approaching  3.5  billion  per  year  and  increasing  at  about  3  percent  per 
year,16  the  importance  of  monitoring  this  segment  of  the  health  care  system  can  be 
easily  understood.  As  medication  use  escalates  and  the  complexity  of  medication 
regimens  increase,  the  importance  of  monitoring  patients  for  medication  induced 
effects  or  medication  errors  is  easily  apparent.  The  Medicare  prescription  benefit 
recently  passed  by  Congress  may  also  increase  medication  use  in  the  outpatient 
setting,  especially  among  the  elderly  by  making  medication  more  affordable  to 
this  vulnerable  population,  a  noble  goal.  While  these  many  factors  are 
contributing  to  increased  use  of  medications,  we  must  find  ways  to  decrease  the 
risk  of  potential  events  related  to  use  of  these  effective  agents.  Computerized 
monitoring  provides  a  reproducible  method  of  detecting  ADEs.  In  addition,  fewer 
resources  are  needed  to  capture  events  compared  to  chart  review.  While  there  is 
an  initial  outlay  of  money  needed  to  allow  proper  programming  of  the  various 
rules,  in  the  long  term  this  process  is  less  expensive  and  it  will  allow  medication 
systems  to  be  safer. 
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While  this  methodology  was  able  to  discover  many  events,  there  are  some 
limitations  to  its  utility.  Though  the  large  majority  of  clinical  notes  from  the  text 
trigger  portion  of  the  monitor  were  able  to  be  accessed,  the  availability  of  some 
specific  classes  of  notes,  such  as  psychiatric  related  notes,  was  very  limited.  As 
such,  the  monitor  may  not  have  been  able  to  capture  events  identified  in  clinics 
that  do  not  routinely  dictate  or  enter  notes  into  the  EMR.  The  ability  of  the  ADE 
monitor  also  relies  on  the  thoroughness  and  accuracy  of  documentation  within  the 
electronic  medical  record.  In  addition  there  is  a  need  to  improve  the  ability  of 
natural  language  processing  to  eliminate  false  positive  results. 

A  limitation  of  this  study  is  that  significant  effort  was  required  to  calibrate  the 
rules  between  the  two  health  care  systems.  However,  this  was  only  because  this 
work  was  done  for  research  purposes.  From  an  operational  standpoint,  these  rules 
can  be  implemented  at  individual  academic  or  non-academic  settings  with 
computerized  lab  and  medication  data5' 6  and  can  be  used  for  real  time  medication 
safety  activities.  However,  as  IT  systems  become  more  widespread,  efforts  to 
standardize  the  rule  sets  and  definitions  so  data  could  be  compared  across 
institutions  would  be  helpful.  Finally,  the  two  sites  used  in  this  study  have  fairly 
advanced  electronic  medical  records  and  information  technology  support 
capabilities.  If  an  EMR  is  not  available,  it  is  extremely  difficult  for 
implementation  of  ADE  monitors  to  occur. 

In  this  report,  we  describe  the  collaboration  of  two  institutions  to  implement 
an  ADE  monitor  in  the  outpatient  setting.  We  found  that  clinically  based  rule  sets 
can  be  implemented  at  different  sites  using  both  text  and  non-text  searching 
methods  and  provide  a  basis  for  improving  quality  by  identifying  adverse  drug 
events  and  medication  errors.  This  monitor  will  be  used  in  the  future  to 
understand  the  effectiveness  of  ADE  quality  improvement  interventions  via 
measurement  of  impact  on  ADEs.  By  using  this  ADE  monitor  we  can  compare 
the  effect  of  our  interventions  without  prohibitive  costs  of  full  chart  reviews.  As 
medication  use  increases — due  to  the  increased  availability  and  effectiveness  of 
medications  in  treating  a  greater  number  of  conditions — we  hope  to  be  able  to 
decrease  the  risk/benefit  ratio  related  to  the  use  of  medications  in  the  ambulatory 
care  setting  by  improved  identification  and  monitoring  of  ADEs. 
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